MUSE (Multidimensional User-Oriented Synthetic Environment): A New Approach to the Human-Computer Interface Using Precognitive Models of Perception

The human mind is capable of absorbing and processing large volumes of information. Most of this processing, however, occurs at a precognitive level, the results of which serve to alert the cognitive mind to areas of potential interest. The Multidimensional User-oriented Synthetic Environment, fluSE is an open-ended software shell that provides a new approach to interacting with computer-based information. By using a real-time, device-independent software design and incorporating both cognitive and experiential models of human perception, fluSE greatly enhances a person's ability to examine, interact with, and understand relationships in complex information space. A jluSE shell may be wrapped around data, models, simulations or even complete programs. Using a design, based on human functionality, it provides tools for the presentation, exploration, navigation, manipulation, and examination of information. Users experience a highly interactive environment, capable of dynamically mapping information into visual, auditory or kinesthetic representations. 1 . I n t r o d u c t i o n Richard Hamming said "The purpose of computing is insight, not numbers." Today, as the Information Age moves forward, computers touch all aspects of our lives. Far from achieving "insight", however, there is often a feeling that we are slowly sinking under the enormous volumes of data. Helping people to explore, question, and understand complex information is an important criterion for future computational environments. Highly interactive human-computer environments can allow practical solutions to some problems far more rapidly than either human or computer operating independently. While a computer excels at performing routine types of computations, it is relatively inefficient at subjective problems such as complex decision making. The mind, on the other-hand, is the most advanced device ever developed for information correlation, pattern recognition, and anomaly detection. Al l of our senses are superbly designed to collect, correlate, and analyze continuous streams of information; to discard irrelevant information at high speeds; and to make intuitive connections. In 1991, Sandia National Laboratories began a project to develop a new approach for interacting with computerbased information. As a result of this effort, a highly interactive, software shell, }luSE (Multidimensional Useroriented Synthetic Environment) has been developed. This shell permits users to create immersive, experiential environments to facilitate the flexible and humanistic exploration multivariate information. 2. The Human Interface Model User interaction with synthetic, immersive environments present an exciting and relatively uncharted area. Enhancing precognitive visual perception requires attention to such factors as head-tracking, peripheral vision, shadow variation, and stereo vision. The auditory sense can be used both as an independent information channel and to enhance other channels. Data sonification can be used to provide auditory queues or baselines against which other occurrences can be compared (e.g., the ticking of a clock vs. the movement of data or objects). If utilized appropriately (and unobtrusively), auditory information representations can be monitored at a precognitive level, leaving the cognitive mind free to independently examine other information. One important concern, however, is achieving balance and not overloading any single sensory channel. In concert with the idea of a humanistically organized software environment, five areas of human-computer interaction have been defined and investigated. They are Exploration, Navigation, Presentation, Interaction, and Examination. These five areas serve to define a functional arena for interactive environments and provide the foundation for device and model independent tools. 2.1 Explorat ion This domain involves techniques for moving within and between various models and representations of information. To facilitate this modality, fluSE creates a virtual craft. This craft can be characterized as a private, extended office that travels with the viewer through the environment. 'Ihe visual representation of the craft provides the user with a personal frame-of-reference within the information space. The craft functionality (e.g., hovercraft, plane, tethercraft) provides the means to explore spatial, temporal, and other dimensions within the information space. £luSE also recognizes time as a potentially independent dimension of exploration. In data where time is a parameter, the system permits the user to control both the speed and direction of time flow. Another means of exploration is by teleportation. MAPLES AND PETERSON 2069 Instantaneous teleportation between marked spatial, temporal and N-dimensional locations is provided. 2.2 N a v i g a t i o n These methods assist users in maintaining both location and reference (e.g., to keep from becoming lost). One such tool is the navigational map that is displayed on the side wal l of the craft. This 'you-are-here' map automatically shows the position of every defined object in the space as wel l as the location and orientation of the craft. Since this map is maintained in real-time, it captures the location and motion of both object and craft positions. N-dimensional markers provide another navigational tool. Markers are stored as N-dimensional reference points. Users may use any portion of a marker's descriptor as a reference information for teleportation. In addition, doorways or portals provide a way of moving between different information spaces. 2.3 P resen ta t i on Presentation refers to ways of representing complex information. These include both visual and auditory representations, as well as tactile, force-feedback and kinesthetic techniques. W i th in the virtual craft, users may invoke cockpit displays, which al low a virtual transparent bubble around the craft to be used for translucent displays of information. Such displays are controlled dynamically and travel wi th the craft. Cockpit displays permit the user to expand the dimensionality of the space by adding additional, nonobtrusive, in format ion displays. Since they can be peripherally located in close virtual proximity to the user, the detail and/or perspectives can be controlled by simply moving the head or body (e.g., leaning closer or turning). 2.4 I n t e r a c t i o n Interaction relates to command and control structures that permit users to communicate, and directly interact, with both the system and application program (e.g., to select options and interact wi th the environment). To facilitate parallel human operations and minimize distracting elements, a humanistically efficient interactive environment should permit the mapping of operations to different areas of the human senses. This permits sub-cognitive input to be maximized and sensory conflict to be minimized. In 3-D environments reading can confl ict wi th spatial concentration. Synthetic speech generation, however, can be used to transform most written or textual information into conversational speech. The auditory and visual channels operate quite harmoniously in parallel, and listening to status or information doesn't normally create visual confl icts. Analogously, the act of requesting status or controlling state functions (e.g., turning things on and off) is often more ef f i ciently performed by voice command rather than by locating buttons (either physically or virtually). 2.5 E x a m i n a t i o n This domain includes capabilities that permit the transformation, correlation, and overlay of information. It typical ly is the most cognit ive of the five functional areas. Examination frequently involves asking questions relating to an information base. One way of posing such questions is by means of the virtual, on-board computer. Mathematical transformation is often used to examine and compress information, particularly when it varies over large dynamic ranges (e.g., semi-log, log or polar plots). This capability also exists in the jluSE environment, but techniques have been conceptually and mathematically extended to encompass more types of information and to in clude non-visual representations. For example, non-linear scaling can be used to require all objects of information are visible, regardless of size or distance. Similarly control of viewer and craft size permits either macroscopic overviews or microscopic examinations (note that such changes may require corresponding adjustments to the interpupilary distance). 3 . S h a r e d E n v i r o n m e n t s Permitting mult iple people to both independently and simultaneously explore the same synthetic environment is a unique aspect of immersive technology. The shared memory architecture of jluSE is designed to enable and extend this capabil i ty. The architecture is based on a distr ibuted, asynchronous communication model. The design supports heterogeneous operation at each site and minimizes the network synchronization and update traffic. The system not only supports multiple users, but permits each to maintain a personalized or private 'cockpit' within the shared space. The cockpits' information displays are individual ly configured and not accessible by others in the environment. 4 . C o n c l u s i o n s fluSE has been developed and tested over the past three years and has been very successfully applied to a wide variety of applications (e.g., C A D models, explosive welding simulations, medical data, seismic in format ion, f in i te element analysis, c ircui t simulat ion, structural analysis, kinematic evaluation, topological data) and has been wrapped around complete programs. The system currently supports f lat screen, stereo, and VR operation (with fu l l head-tracking), voice recognition, sound synthesis, data sonification, and various commercial interactive devices. The fact that all jluSE applications run on any configuration or subset of these hardware devices (down to a monitor, keyboard and mouse) illustrates the hardware independence of the design. The humanistic basis of the software design has proven enormously effective, both in enabling people to interact with computer-based information and in speeding up the analysis and decision making process. One of the greatest surprises in applying fUuSE to different applications has been the frequency wi th which it permitted people to discover important aspects of their own applications of which they were previously unaware. fiuSE (pronounced 'muse') is trademarked; patents are pending. Work performed at Sandia National Laboratories is supported by the U.S. Department of Energy under contract DE-AC04-94AL85000